Method and apparatus for recognizing handwritten words

ABSTRACT

A method for the angle-independent recognition of handwritten objects including: a) receiving a handwritten stroke; b) normalizing the stroke; c) matching the normalized stroke against a stroke database to obtain at least one character part interpretation; and d) recognizing a handwritten object using one or more of the character part interpretations. Preferably, the step of normalizing the stroke includes the step of normalizing an angle of the stroke by first determining a stroke angle and then rotating the stroke by the stroke angle until the stroke is at a normalized position. An apparatus for the angle independent recognition of handwritten objects includes: a) a central processing unit (CPU); b) a pointer mechanism coupled to the CPU for providing a handwritten stroke to the CPU; c) a mechanism for normalizing the strokes provided by the pointer mechanism; d) a mechanism for matching the normalized strokes against a stroke database to obtain a number of character part interpretations; and e) a mechanism for recognizing a handwritten object using the character part interpretations. Preferably, the mechanism for normalizing the strokes includes a mechanism for detecting stroke angles of the strokes, and a mechanism for rotating the strokes by the stroke angles to create normalized strokes. The mechanism for recognizing the handwritten object preferably includes a controller, a number of recognition domains coupled to the controller, and an arbiter coupled to the controller for resolving multiple interpretations of the object.

CROSS-REFERENCE TO RELATED APPLICATIONS

This is a continuation of application Ser. No. 08/068,443 filed May 27,1993, now abandoned, which is, a continuation-in-part of the followingco-pending U.S. patent applications:

1. "Recognition Architecture and Interface", of Beernink et al., U.S.Ser. No. 07/889,216, filed May 27, 1992 and assigned to Apple Computer,Inc. now abandoned and;

2. "Method and Apparatus for Computerized Recognition", of Pagallo etal., U.S. Ser. No. 08/001,123, filed Jan. 5, 1993 and assigned to AppleComputer, Inc., now abandoned.

The disclosures of each of these parent applications are herebyincorporated herein by reference in their entirety.

BACKGROUND OF THE INVENTION

This invention relates generally to handwriting recognition, and moreparticularly to handwriting recognition in a pen-based computer system.

A pen-based computer system typically comprises a small, hand-heldcomputer where the primary method for inputting data includes a "pen" orstylus which "writes" upon the screen of the computer system. The screenis a dual-function display assembly which serves as both an input deviceand an output device. When operating as an input device, the displayassembly senses the position of the tip of a stylus on the viewingscreen and provides this positional information to the computer'scentral processing unit (CPU). Some display assemblies can also sensethe pressure of the stylus on the screen to provide further informationto the CPU. When operating as an output device, the display assemblypresents computer-generated images on the screen.

The dual-function display assemblies of pen-based computer systemspermit users to operate the computer as a computerized notepad. Forexample, graphical images can be input into the pen-based computer bymerely moving the stylus on the surface of the screen. As the CPU sensesthe position and movement of the stylus, it generates a correspondingimage on the screen to create the illusion that the stylus is drawingthe image directly upon the screen, i.e. that the stylus is "inking" animage on the screen. Besides serving as a notepad, pen-based computerscan provide a number of useful functions, such as serving as an addressbook, an appointment calendar, a to-do list, etc.

Ink on the screen of a pen-based computer system is typically stored asa simple bit-map. Essentially, the only knowledge that the computersystem has of the inked image is that certain pixels of the display areto be activated to create the inked image. Therefore a note, such as"Meet Bob Tuesday" has no meaning to the system other than certainpixels of the screen are to be turned on or off to create the inkedimage of the words.

It is often desirable to perform some level of recognition on the inkedobjects formed on the computer screen. In this way, additional meaningcan be attached to the inked objects allowing the computer to manipulatethe objects in a more intelligent fashion. For example, a commonoperation for pen-based computer systems is handwriting recognition. Inthis way, the inked object "Meet Bob Tuesday" can be "recognized" ascomprising three words separated by spaces where each word comprises anumber of letters. The recognized words and letters can be convertedinto an appealing font, and can be used and manipulated as words andletters rather than as bits in a bit-map.

There are a great many methods for recognizing handwriting known in theprior art. However, these recognition methods tend to be narrowlyfocused on a particular type of recognition task. For example, there areprior art recognizers for recognizing alphanumeric inputs, forrecognizing graphical inputs, etc. If a recognizer of the prior art doesnot recognize a particular type of object that object is typicallytreated as ink.

Most handwriting recognizers require that a user enter handwritteninformation in a constrained manner. For example, many handwritingrecognizers require that letters be entered in "block" form, rather thanin a cursive fashion. Other word recognizers require that words beentered in particular fields or areas of the screen. Most handwritingrecognizers require that words be written in a substantially horizontalmanner from left to right.

These constraints limit the flexibility of handwriting recognizers, andtakes some of the creativity and enjoyment out of the use of a pencomputer system. It would be desirable to have a method and apparatusfor recognizing handwritten words which would allow the characters ofthe word to be written in a free-form fashion at any angle, in anydirection or order, and virtually anywhere on the screen of a pencomputer system.

SUMMARY OF THE INVENTION

A character based word recognizer in accordance with the presentinvention allows words to be entered into a pen computer system atvirtually any angle, in any direction, and virtually anywhere on thescreen of the computer system. This permits, for example, words to bewritten horizontally, vertically, or any angle in between. Also, lettersmay be written at an angle to the baseline of the word of which theyform a part. The word recognizer of the present invention also permitsthe letters of a word to be written in any order, e.g. the word "the"can be entered by first entering an e, then entering an h to the left ofthe e, and then entering a t to the left of the h.

A method for the angle independent recognition of handwritten objects inaccordance with the present invention includes the steps of: a)receiving a handwritten stroke; b) normalizing the stroke; c) matchingthe normalized stroke against a stroke database to obtain at least onecharacter part interpretation; and d) recognizing a handwritten objectusing one or more of the character part interpretations. The step ofnormalizing the stroke preferably comprises normalizing an angle of thestroke to create an angle-normalized stroke. By angle-normalizing astroke prior to matching it into a stroke database, the angle andorientation of the stroke relative to a baseline of a word or to anarbitrary (x,y) coordinate system becomes much less relevant indetermining the nature of that stroke. Preferably, the step ofnormalizing also includes the step of normalizing an aspect ratio of thestroke to avoid recognition errors due to distortion.

An apparatus for the angle independent recognition of handwrittenobjects includes: a) a central processing unit (CPU); b) a pointermechanism coupled to the CPU for providing a handwritten stroke to theCPU; c) a mechanism for normalizing the strokes provided by the pointingmechanism; d) a mechanism for matching the normalized strokes against astroke database to obtain a number of character part interpretations;and e) a mechanism for recognizing a handwritten object using thecharacter part interpretations. The mechanism for normalizing thestrokes preferably includes a mechanism for detecting stroke angles, anda mechanism for rotating the strokes by the stroke angles to createnormalized strokes. The mechanism for normalizing strokes can furtherinclude a mechanism for detecting the aspect ratios of the strokes, anda mechanism for normalizing the aspect ratios of the strokes if theaspect ratios are not within predetermined limits. The mechanism formatching the normalized strokes preferably includes a stroke templatedatabase having a number of stroke template entries corresponding to anumber of character parts. The mechanism for recognizing a handwrittenobject preferably includes a controller, a number of recognition domainscoupled to the controller, and an arbiter coupled to the controller forresolving multiple interpretations of the object.

These and other advantages of the present invention will become apparentupon a reading of the following descriptions and a study of the variousfigures of the drawing.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram of a computer system in accordance with thepresent invention;

FIG. 2 is a top plan view of the screen, case, and keypad of thecomputer system of FIG. 1;

FIG. 3 is a block diagram of a word recognition system in accordancewith the present invention;

FIG. 4 is a flow diagram illustrating a computer implemented process ofthe word recognizer illustrated in FIG. 3;

FIG. 5 is a block diagram illustrating a recognition domain of FIG. 3;

FIG. 6 is a flow diagram illustrating a "NEEDS GROUPING" call to acharacter part recognition domain of FIG. 4;

FIG. 6(a) is a pictorial representation of a STROKE UNIT;

FIG. 6(b) is a pictorial representation of a CHARPART UNIT of thepresent invention;

FIG. 7 is a flow diagram illustrating a "NEEDS CLASSIFICATION" call to aCHARPART recognition domain in FIG. 4;

FIG. 8 is a flow diagram illustrating the "NORMALIZE STROKE" step ofFIG. 7;

FIG. 9 is a flow diagram illustrating the "NORMALIZE ANGLE" step of FIG.8;

FIGS. 9a-9d are used to illustrate the process 162 of FIG. 9;

FIG. 10 is a flow diagram illustrating the "NORMALIZE BOUNDING BOX" stepof FIG. 8;

FIG. 11 is a diagram of the database structure used in the "MATCHNORMALIZED STROKE" step of FIG. 7;

FIG. 11(a) is a pictorial representation of the data structure of acharacter part interpretation "CHARPARTINT" produced from the datastructure of FIG. 11;

FIG. 11(b) is a pictorial representation of how the character partdescriptions are derived in FIG. 11;

FIG. 12 is a flow diagram illustrating the "NEEDS GROUPING" call to aword recognition domain in FIG. 4;

FIG. 12(a) is a pictorial depiction of a WORD UNIT produced by theprocess of FIG. 12;

FIG. 13 is a pictorial representation of the "NEEDS CLASSIFY" call tothe word recognition domain of FIG. 4;

FIG. 14 is a flow diagram of a process undertaken by the wordrecognition domain after the "NEED CLASSIFY" call in FIG. 4;

FIG. 14(a) is a pictorial depiction of a "t" produced at an angle; and

FIG. 14(b) is a depiction of a data structure produced by the process ofFIG. 14 to provide word interpretations for handwritten words.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

The present invention is well suited for pointer based computer systemssuch as the pen-based, stylus-based and mouse driven systems that arecurrently popular. For the purposes of illustration, the invention willbe described in connection with a pen-based system.

As shown in FIG. 1, a pen-based computer system 10 in accordance withthe present invention includes a central processing unit (CPU) 12, readonly memory (ROM) 14, random access memory (RAM) 16, input/output (I/O)circuitry 18, and a display assembly 20. The pen-based computer system10 may also optionally include a mass storage unit 22 such as a diskdrive unit or nonvolatile memory such as flash memory, a keypad 24, anda clock 26.

The CPU 12 is preferably a commercially available, single chipmicroprocessor. While CPU 12 can be a complex instruction set computer(CISC) chip, it is preferable that CPU 12 be one of the commerciallyavailable, reduced instruction set computer (RISC) chips which are knownto be of generally higher performance than CISC chips. CPU 12 is coupledto ROM 14 by a unidirectional data bus 28. ROM 14 contains the basicoperating system for the pen-based computer system 10. CPU 12 isconnected to RAM 16 by a bi-directional data bus 30 to permit the use ofRAM 16 as scratch pad memory. ROM 14 and RAM 16 are also coupled to CPU12 by appropriate control and address busses, as is well known to thoseskilled in the art. CPU 12 is also coupled to the I/O circuitry 18 bybi-directional data bus 32 to permit data transfers with peripheraldevices.

I/O circuitry 18 typically includes a number of latches, registers anddirect memory access (DMA) controllers. The purpose of I/O circuitry 18is to provide an interface between CPU 12 and such peripheral devices asdisplay assembly 20, mass storage 22, and the keypad 24.

Clock 26 provides clock data and is typically coupled to a port of CPU12 by a data line 34. The clock data is used to time various functionsand events relating to the computer system 10. The clock 26 can beeliminated and the clock function replace by a software clock running onCPU 12, but this tends to be a wasteful use of CPU processing power.

Display assembly 20 of pen-based computer system 10 is both an input andan output device. Accordingly, it is coupled to I/O circuitry 18 by abi-directional data bus 36. When operating as an output device, thedisplay assembly 20 receives data from I/O circuitry 18 via bus 36 anddisplays that data on a suitable screen. The screen for display assembly20 is preferably a liquid crystal display (LCD) of the type commerciallyavailable from a variety of manufacturers. The input device of displayassembly 20 is preferably a thin, clear membrane which covers the LCDdisplay and which is sensitive to the position of a stylus 38 on itssurface. The membrane covering the screen therefore serves as an input"tablet." These position sensitive membranes are also readily availableon the commercial market. Combination display assemblies such as displayassembly 20 which include both the LCD and the input membrane arecommercially available from such vendors as Scriptel Corporation ofColumbus, Ohio.

The keypad 24 can comprise an array of switches. In the presentembodiment, the keypad 24 comprises "button" areas which overlie thebottom edge of the membrane which covers the LCD display. When the"buttons" are depressed, the membrane senses the pressure andcommunicates that fact to the CPU 12 via I/O 18.

Other types of pointing devices can also be used in conjunction with thepresent invention. While the method of the present invention isdescribed in the context of a pen-based system, other pointing devicessuch as a computer mouse, a track ball, or a tablet can be used tomanipulate a pointer on a screen of a general purpose computer.Therefore, as used herein, the terms "pointer", "pointing device","pointing means", and the like will refer to any mechanism or device forpointing to a particular location on a screen of a computer display.

Some type of mass storage 22 is generally considered desirable. Massstorage 22 can be coupled to I/O circuitry 18 by a bi-directional databus 37. However, the mass storage 22 can be eliminated by providing asufficient amount of RAM 16 to store user application programs and data.In that case, the RAM 16 could be provided with a backup battery toprevent the loss of data even when the pen-based computer system 10 isturned off. However, it is generally desirable to have some type of longterm storage 22 such as a commercially available miniature hard diskdrive, nonvolatile memory such as flash memory, battery backed RAM,PC-data cards, or the like.

In operation, information is input into the pen-based computer system 10by "writing" on the screen of display assembly 20 with a stylus 38.Information concerning the location of the stylus 38 on the screen ofthe display assembly 20 is input into the CPU 12 via I/O circuitry 18.Typically, this information comprises the Cartesian (i.e. x & y)coordinates of a pixel of the screen of display assembly 20 over whichthe tip of the stylus is positioned. Commercially available combinationdisplay assemblies such as the aforementioned assemblies available fromScriptel Corporation include appropriate circuitry to provide the styluslocation information as digitally encoded data to the I/O circuitry ofthe present invention. The CPU 12 then processes the data under controlof an operating system and possibly an application program stored in ROM14 and/or RAM 16. The CPU 12 next produces data which is output to thedisplay assembly 20 to produce appropriate images on its screen.

In FIG. 2, the pen-based computer system 10 is shown housed within agenerally rectangular enclosure 40. The CPU 12, ROM 14, RAM 16, I/Ocircuitry 18, mass storage 22, and clock 26 are preferably fullyenclosed within the enclosure 40. The display assembly 20 is mostlyenclosed within the enclosure 40, but a viewing screen 42 of the displayassembly is exposed to the user. As used herein, the term "screen" willrefer to the portion of the display assembly 20 which can display animage that can be viewed by a user. Also accessible to the user is thekeypad 24.

Upon power-up, pen based computer system 10 displays on screen 42 aninitial note area N including a header bar B and a number of guidelines44. The header bar B preferably includes the date of creation 46 of thenote N, a note number 48, and a "toolbox" button 50. The optionalguidelines 44 aid a user in entering text, graphics, and data into thepen-based computer system 10.

The keypad 24 preferably includes a number of dedicated function"buttons" 52 and a pair of scroll "buttons" 54A and 54B. The operationof the scroll buttons 54A and 54B, and other aspects of computer system10 are discussed in greater detail in co-pending U.S. patent applicationSer. No. 07/868,013, filed Apr. 13, 1992 on behalf of Tchao et al.,entitled "Method for Manipulating Notes on a Computer Display", andassigned to the same assignee as the present application, the disclosureof which is hereby incorporated herein in its entirety.

The screen illustrated in FIG. 2 is referred to as the "notepad", and isan application program running under the operating system of the penbased computer system 10. In this preferred embodiment, the notepad is aspecial or "base" application which is always available beneath higherlevel applications. The notepad application, like other applications,run within a window, which in this instance comprises the entire viewingscreen 42. Therefore, as used herein, a "window" is the entire screen orany portion of an entire screen which is dedicated to a particularapplication program.

A status bar 56 is provided at the bottom of the notepad application.The status bar 56 is provided with a number of active areas including areal time clock 58, a view button 60, a font button 62, a formulasbutton 64, a text button 66, a graphics button 68, and a nib button 70.U.S. patent application Ser. No. 07/976,970 filed Nov. 16, 1992 onbehalf of Foster et. al, entitled "Status Bar for Application Windows"and assigned to the assignee of the present invention describes how tomake and use the status bar, and is incorporated herein by reference inits entirety.

In FIG. 3, the structure of the recognition system 72 of the presentinvention is illustrated. The recognition system 72 includes acontroller 74, recognition areas 76, a database 78, recognition domains80, and an arbiter 82. The controller 74 has, as inputs, stroke units,other types of units (optional), and area registration information. Thearbiter 136 has an input for returned units from an application program,and has outputs for stroke units and recognized units. The constructionand operation of recognition system 72 is described in greater detail incopending Parent Patent Application U.S. Ser. No. 08/001,123, filed Jan.5, 1993 previously incorporated by reference.

FIG. 4 is a flow diagram illustrating the operation of the controller 74of FIG. 8. A computer implemented process 84 begins at 86 and, in a step88, it is determined whether the unit comprises a new stroke. If itdoes, the AREALIST of the stroke unit is filled-in in a step 90. In astep 92, it is determined whether any unit needs grouping. If it does,each unit is passed to the recognition domains in a step 94, and eachnew group unit is moved into a "classification pool" in a step 96.

In step 98, it is determined whether any unit needs classification. Ifit does, each unit is passed to the recognition domains in a step 100,and a step 102 places each classified unit into a "grouping andarbitration pool." A step 104 determines whether arbitration isrequired, and if it is, each unit is arbitrated in a step 106. If noarbitration is needed, or after arbitration is completed, processcontrol returns to step 88.

The process 84 of FIG. 4 and its interactions with the classificationpool and the grouping an arbitration pool is described in detail incopending U.S. patent application Ser. No. 08/001,123, supra.

FIG. 5 illustrates a recognition domain 108 which may be one of therecognition domains 80 of FIG. 8. This recognition domain 108 is an"object" including a data portion 110 and a procedures portion 112. Theprocedures portion 112 communicates with the controller 74 in adialogue-like fashion. The procedures portion 112 has access to the dataportion 110, which includes such data as the type of units recognized bythe recognition domain 108, dictionaries, and domain parameters. Eachdomain 80 has both grouping and classification capabilities.

In the present invention, a character part ("CHARPART") domain and aword ("WORD") domain are provided, where words are deduced directly fromcharacter parts. In alternate embodiments of the present invention, acharacter ("CHARACTER") domain can be provided to deduce characters fromthe CHARPART domain, and such that words can be deduced from theCHARACTER domain. However, it has been found that better recognition ispossible with the present invention by deducing words directly fromcharacter parts.

The term "object" has been used extensively in the precedingdiscussions. As is well known to software developers, an "object" is alogical software unit comprising data and processes which give itcapabilities and attributes. For example, an object can be queried as toits type and can return such data as the number of words that itcontains. Objects can contain other objects of the same or of adifferent type. Objects can also be used to project images on a screenaccording to their object type. There are many well know texts whichdescribe object oriented programming. See, for example, Object OrientedProgramming for the Macintosh, by Kurt J. Schmucher, Hayden BookCompany, 1986. Since the definition of "object" has been somewhatloosely used in the industry, we will refer to a certain types ofobjects as "units." More particularly, we will refer to STROKE UNITS,CHARPART UNITS, and WORD UNITS, it being understood that these units are"objects" of a particular type and format.

A STROKE UNIT 124 as received by step 118 of FIG. 6 is shown in FIG. 6a.Like all computerized objects, a STROKE UNIT 124 includes a data portion126 and a procedures portion 128. The data portion 126 is broken intothree sections, namely, generic data 130, subunit data 132, andinterpretation data 134. The generic data section includes a variety ofinformation such as the type of unit (which in this case is a strokeunit) certain flags (such as Visible and Used), the bounding box for thestroke as defined by x_(min), y_(min), x_(max), y_(max), the AREALIST,the minimum stroke identification of the unit (minSTROKEID) the maximumstroke identification of the unit (maxSTROKEID), and various typespecific information such as the number of points in the stroke. Thesubunit data field 132 includes information concerning any subunits ofthe STROKE UNIT 124. The interpretation data 134 includes the array ofdata points A (x, y, t, p) which is graphically illustrated in FIG. 6.Of course, in the computer system 10, these data points are stored as anarray or queue of binary numbers. Preferably, a queue A extends from thefirst data point (x₀, y₀, t₀, p₀) to the n+1 data point, (x_(n), y_(n),t_(n), p_(n)). The procedures portion 128 of the stroke unit providesthe necessary methods and processes for the STROKE UNIT 124 to operateas an autonomous object, as is well known to those skilled and the art.

FIG. 6b is used to illustrate a CHARPART UNIT produced by step 120 ofFIG. 6. A CHARPART UNIT 136 includes a data portion 138 and a proceduresportion 140. The data portion 138 includes a generic section 142, asubunit section 144, and an interpretation section 146. The genericsection includes data types which will be found in all unit structures.For example, the generic section will include the type (which in thiscase is CHARPART), flags such as sleep and claimed, an AREALIST, abounding box, minSTROKEID, maxSTROKEID, and type specific information.The subunit section 144 will contain subunits of the unit 136, which inthis instance is STROKEUNIT1. The interpretation data section 146 is alist of the interpretations for the unit 138 with their rankings. Notall possible interpretations are shown in this illustration. The lowerthe rank number of the interpretation, the higher the level ofconfidence that the object has been recognized correctly. The proceduresportion 140 include methods and processes which allow the unit tooperate as an object.

As used herein, a STROKE and a CHARPART UNIT might be used somewhatinterchangeably. Each CHARPART UNIT contains but a single STROKE UNIT.Therefore, at times a rotation of a stroke forming a part of a STROKEUNIT which a sub-unit of a CHARPART UNIT will be referred to as arotation of a CHARPART, and vice versa. In actuality, it is a privatecopy of the stroke owned by the CHARPART UNIT that is rotated, since thestroke unit itself should not be modified.

FIG. 7 illustrates a computer-implemented process 148 undertaken by acharacter part recognition domain 100 after a "NEEDS CLASSIFICATION"call 98 of FIG. 4. Process 148 begins at 150 and, in an optional step152, the incoming stroke unit is filtered. This filtering process can,for example, take the form of dividing the incoming stroke into sixteenequally spaced segments which replaces the multiplicity of randomlyspaced segments of an incoming raw stroke. Such "filtering" helpsoptimize the process, but is not an indispensable part of the presentinvention. Next, in a step 154, the incoming stroke is normalized. Otherimplementation features might be extracted from the normalized stroke atthis time. Finally, in a step 156, the normalized stroke is matched intoa stroke data base to obtain a list of charter part interpretations. Theprocess 148 is then completed as indicated at 158.

FIG. 8 is a flow diagram illustrating the "NORMALIZE STROKE" step 154 ofFIG. 7 in greater detail. Process 154 begins at 160 and, in a step 162,the angle of the character part unit is normalized. Next, in a step 154,the bounding box of the character part unit is also normalized. Thisbounding box normalization step 164 is optional, and again is providedas an optional optimization technique. The process 154 is then completedas indicated at 166.

FIG. 9 is a flow diagram illustrating the "NORMALIZE ANGLE" step 162 ofFIG. 8 in greater detail. The process 162 begins at 168 and, in a step170, the starting and ending points of the stroke are determined. Next,in a step 172, the bounding box of the stroke is determined. As is wellknown to those skilled in the art, the "bounding box" is the minimumrectilinear box which fully encloses the stroke. Next, in a step 174,the center of the stroke is located. A subsequent decision step 176compares the distance between STOPPOINT and STARTPOINT (the starting andending points of the stroke) and compares them to a distance which is apercentage of the length of the stroke, e.g., one-third of the width ofthe bounding box of the stroke. In other words, step 176 calculatesSTOPPOINT minus STARTPOINT and, if the difference is greater than orequal to the distance d%, then step 178 is performed. The distance d% isderived by the size of the bounding box. Step 178 determines the angle φbetween STARTPOINT and STOPPOINT relative to the vertical axis. Asexplained previously, the pixels of the screen 42 are arranged in aCartesian-coordinate system such that the vertical axis is parallel to a"Y" axis of the screen 42. Next, in a step 180, CHARPART is rotated by φaround its center such that the STARTPOINT of the stroke is located atthe top of CHARPART. The process is then completed as indicated at 182.

If the result of the decision step 176 is in the affirmative, a step 184finds a FARPOINT in the CHARPART which is farthest from STARTPOINT. Thiswould be the case with a substantially circular item. Next, in a step186, the STOPPOINT is set to the FARPOINT. Thereafter, steps 178 and 180are performed and the process is again completed at 182.

FIGS. 9a-9d illustrate the process 162 of FIG. 9. If, for example, astroke unit enters the process 162 having a stroke 188, its starting andending points are determined at 190 and 192. The bounding box 194 isthen determined by step 172 and the center 196 of the stroke 188 is thendetermined by step 174. Since STOPPOINT 192 minus STARTPOINT 190 is muchgreater than one-third the distance between those points (e.g. d%), step176 determines the process control should be turned over to step 178. Instep 178, the angle φ between STARTPOINT 190 and STOPPOINT 192 relativeto a vertical axis V is determined, and in a step 180 the stroke ofCHARPART is rotated by φ around the center 196 such that the STARTPOINT190 is at the top of the CHARPART.

In FIG. 9c a CHARPART having a stroke 198 is entered and the startingand ending points are determined by step 170 to be STARTPOINT 200 andSTOPPOINT 202. Again, the bounding box 204 is determined by step 172 andthe center of the stroke 206 is determined by step 174. In thisinstance, step 176 determines that the distance between STOPPOINT 202and STARTPOINT 200 is less than one-third of the width of the boundingbox and, therefore, process control is turned over to step 184. Next, ina step 184, the point FARPOINT is determined by comparing all of thepoints of the stroke 198 to STARTPOINT 200 to determine which is thefarthest away from the STARTPOINT 200. This point FARPOINT should befairly close to the center point 206. Next, in a step 186, the stoppoint is set to the value of FARPOINT. In step 178 the angle φ betweenSTARTPOINT and STOPPOINT relative to the vertical axis v is againcalculated, and in step 180 the CHARPART is rotated by φ around itscenter such that the STARTPOINT 200 is at the top of the CHARPART.

This rotation process illustrated in FIGS. 9 and 9a-9d is fundamental toproviding a word recognizer which is angle and direction independent.Since all character parts are rotated or normalized by the process 162,the actual angle that they are presented on the screen does not affectthe recognition process.

FIG. 10 illustrates the optional step 164 of FIG. 8 in further detail.In the step 164, the aspect ratio of the bounding box is determined and,if it is too severe, the CHARPART is scaled appropriately to bring theaspect ratio of the bounding box within a predetermined criteria. It hasbeen determined empirically that if the aspect ratio defined as thewidth of the bounding box divided by the height of the bounding box isgreater than 3 or less than 1/3 that the word recognition process can beimproved by bringing the aspect ratio within these limits. Morespecifically, the process 164 begins at 208 and, in a step 210, theaspect ratio of the bounding box is calculated. As mentioned previously,this aspect ratio is simply determined by dividing the width of thebounding box in the X direction by the height of the box in the Ydirection, i.e. x/y. Next, in a decision step 212, it is determinedwhether the aspect ratio is greater than 3. If not, a step 213determines whether the aspect ratio is less than 1/3, i.e. less than0.333. If not, the aspect ratio is within reasonable limits and theprocess is completed at 214. If the aspect ratio is less than 0.333, theCHARPART is scaled in an X direction by a factor of the reciprocal ofthe aspect ratio, and the process is completed at 214. If the aspectratio is greater than 3, a step 215 scales the CHARPART in a Y directionby the aspect ratio and the process is again completed at 214.

FIG. 11 illustrates the data structure used by the step 156 "MATCHNORMALIZE STROKE" of FIG. 7. The database 220 includes a stroke templateportion 222, a character description portion 224 and a character partdescription portion 226. A stroke template portion 222 includes a numberof stroke template entries 228a, 228b, etc. for matching against strokeparts. The stroke template can be either stroke or feature based,although a stroke based template is preferred in this present invention.Methods for implementing stroke and feature based templates are wellknown to those skilled in the art.

The stroke template 222 includes a plurality of pointers 230 which pointto various locations in a character description 224. For example, forstroke template entry 228a, which is a vertical line, a plurality ofpointers 230 point to character description areas 232a, 232b, etc. Inother words, a match against stroke template entry 228a might be thefirst stroke of the letter "t", the second stroke of the letter "t", orthe second stroke of a letter "d". Stroke template entry 228a thereforepoints to a character description 232a for the symbol "f" and acharacter description 232b for the symbol "d". Of course, there are manyother character descriptions that can apply to stroke template entry228a, the two illustrated being merely examples. Likewise, stroketemplate entry 228b, which resembles a lowercase "c" points to characterdescription 232b, because it can be the first stroke of the letter "d".The possible stroke number and symbol linking a stroke template entry toa character description is illustrated, by way of example, on some ofthe pointers 230. The sum of these pointer descriptions comprise theCHARPART interpretations for the CHARPART UNIT described previously.

Each of the character descriptions 232a, 232b, etc. also have a pointer234 into the character part descriptions 226. More specifically,character part descriptions 226 include a plurality of areas 236a, 236b,etc. As an example, the character description 232a is of the symbol orletter "t". The number of strokes associated with this symbol is 2. Alsoprovided in character description 232a is the position of the symbol andits stroke index. A pointer 234 connects the character description 232ato a first character part description 236a. Since the number of strokesof the symbol "t" is 2, both character part description 236a and 236bare pointed to by the character description 232a.

The character part description 236a includes a number of pieces ofinformation. Character part description 236a includes the informationCENTERX which is the ideal center position for stroke 1 of a "t", Δxwhich is the x deviation from this ideal center position by the stroke,CENTERY which is the ideal position of the center in the y direction forthe first stroke of the letter "t", Δy which is the deviation of thestroke from this ideal, the ideal angle of this stroke (which for stroke1 of a "t" is 0°), and the deviation Δq between this ideal and theactual angle of the first stroke. Data concerning stroke 2 is likewiseprovided in character part description 236b.

An important optimization is "pre-rotating" the stroke template entries228a, 228b, etc. prior to matching them against the incoming CHARPART.Since the CHARPARTS have already been pre-rotated, a pre-rotated stroketemplate entry will cause rapid, angle independent matching of theCHARPART to the stroke template entry. This is very advantageous overthe prior art which often requires multiple templates at various anglesto represent the same stroke parts.

FIG. 11a is used to describe the data structure of the CHARPARTinterpretations for one of the interpretations (1,t). This CHARPARTinterpretation is one part of the interpretation field of the CHARPARTUNIT. As can be seen in FIG. 11a, various parameters are stored withinthe CHARPART interpretation data structure. More specifically, thevariable stroke order indicates that the order of this stroke is 1, i.e.this is the first stroke to be entered of a "t". The variable characteris equal to "t", and a score is assigned in the range of 0 to 1000points depending upon the certainty of the system as to whether thisparticular character part is, indeed, the first stroke of the "t". Thenumber of strokes (NSTROKES) of this letter "t" is 2, and the angle ofthis first stroke of the "t" is 0°.

FIG. 11b illustrates the character part descriptions 226 of FIG. 11 ingreater detail. Each character fits within a bounding box having a lowerleft-hand coordinate of (0,0) in having an upper right-hand coordinateof (100,100). The character "t" is within this bounding box. Stroke 1 isindicated as S1 and stroke 2 is indicates as S2. The center of stroke S1is, therefore, (50,50) while the center of stroke S2 is about (50,70).The angle of stroke S1 is 0°, while the angle of stroke S2 is -90°. Theactual character parts are matched against this idealized version of thecharacter parts to determine the appropriate Δx,Δy, and Δq values.

FIG. 12 is a flow diagram illustrating the process within a wordrecognition domain when it receives a "NEEDS GROUPING" call from step 92of FIG. 4. A computer implemented process 238 starts at 240 and, in astep 242, it is determined whether there is an existing word unit. Ifnot, a new word unit is made and the CHARPART UNIT is added to it as itssub-unit. The process is then completed as indicated at 246. If it isdetermined by step 242 that there is an existing word unit, it isdetermined in a step 248 whether this new CHARPART UNIT belongs to theword unit. If not, the WORD UNIT is ready to classify and is so markedin a step 252, and a new WORD UNIT is created for the CHARPART UNIT in astep 244. If step 248 determines that a CHARPART UNIT does belong to theWORD UNIT, a step 250 adds the CHARPART UNIT as a subunit of theexisting WORD UNIT and the process is completed at 242. A WORD UNIT 254created by the process 244 of FIG. 12 is illustrated in FIG. 12a.

As before, this WORD UNIT is an object having a data portion 256 and aprocedure portion 258. The data portion 256 includes a generic section260, a subunit section 262, and an interpretation section 264. As can beseen, the subunit section 262 includes a number of CHARPARTS along withtheir representative CHARPART interpretations. Section 264 includesinterpretations for the word which here give a rank of 110 for the word"car" and a rank of 420 for the word "uar".

FIG. 13 is used to illustrate the process undertaken by the wordrecognition domain when a "NEEDS CLASSIFICATION" call is made by step 98of FIG. 4. In this case, it is tempting to classify the word unitcorresponding to the word "the". As can be seen in FIG. 13, therecognition domain takes the four CHARPARTS of the word "the", namelythe first stroke of the letter "t", the second stroke of the letter "t",the stroke comprising the letter "h", and the stroke comprising theletter "e" and creates a list of CHARPARTS interpretations for each ofthese CHARPARTS. These are provided in the subunit information sectionof the WORD UNIT.

FIG. 14 illustrates the actual process of the word recognition domainwhen the "NEEDS CLASSIFICATION" step 98 of FIG. 4 makes a call to theword recognition domain. The computer implemented process 260 begins at262 and, in a step 264, a first NODE (0) is initialized. Next, in aniterative loop step 266, a variable STATE is initialized to zero, and iscompared to a variable NCHARPARTS, which corresponds to the number ofCHARPARTS in the word unit. If STATE is less than NCHARPARTS, a step 268obtains all valid transitions from all nodes at the current STATESnumber using the angle information. Next, in a step 270, for all validtransitions new NODES are created at STATE+1. Next, in a step 272, angleinformation is accumulated at each new NODE. Finally, in a step 274, thescores for all new NODES are updated using the accumulated angleinformation of step 272. The value of the variable STATE is theniterated by one and the loop 268-274 is repeated while STATE is lessthan or equal NCHARPARTS. At the completion of iterative loop step 266,a step 276 obtains NODES at the highest STATE number (which isNCHARPARTS), and picks the ones with the highest scores. These highestscored terminal nodes are put into the word interpretation section ofthe word unit in a step 278, and the process is completed at 280.

In FIG. 14a, the step 272 of FIG. 14 is explained in greater detail.Here, a pair of strokes S1 and S2 which could comprise either a "t" or a"x" is stored as CHARPARTS 1 and CHARPARTS 2 in the subunit section ofthe word unit. The interpretation of the stroke S1 is that it could be a1/2 of "t" at an angle of -5°. An interpretation of stroke S2 is that itcould be stroke 2/2 of "t" at -10°. The angles of these CHARPARTinterpretations are taken relative to the ideal angle, which in the caseof stroke S1 is 0°, and which in the case of stroke S2 is -90°. Theseangle errors can be averaged to provide an interpretation of a "t" at anangle of -7.5°. It is this kind of angle information that is accumulatedby step 272 and which is used in the scoring process 274. Obviously, thecloser that the error angle Δq is to 0°, the higher the level ofconfidence that this is the correct character.

FIG. 14b illustrates the data structure developed by the process of 260.NODE(0) is an initialized node, and has pointers to two entries, i.e.NODE(1) and NODE(2). A first entry is an interpretation for the firstCHARPART which is that it is 1/1 of "i" at 0°. The second interpretationis that it is 1/2 of "t" at 0°. STATE(2) also has two interpretationsfor the second CHARPART, namely that it is 1/2 of "t" at -90°, or 2/2 of"t" at 0°. At NODE(3), the first string of interpretations has come to adead-end, i.e. there is no possible interpretation for the CHARPARTSalong this first row. However, NODE(4) has two possible interpretationsinto NODE(5) and NODE(6), namely, that the third CHARPART is either 1/1of "h" at 0° or 1/1 of "n" at 20°. Finally, at STATE(4), two possibleinterpretations branching off the two valid nodes of NODE(5) and NODE(6)are that the fourth CHARPART is 1/1 of "e" at 0° or 1/1 of "e" at 0°.Therefore, at NODE(7) and NODE(8), which are "terminal nodes", there aretwo possible interpretations for the CHARPARTS 1-4. A firstinterpretation is that the CHARPARTS 1-4 form the word "the", and asecond interpretation is the CHARPARTS 1-4 form the word "tne". Thesetwo possible interpretations are weighted using the accumulated angleinformation (in addition to other available information such ascharacter size, baseline, score, etc.) and, hopefully, the correctchoice "the" is given the best (i.e. lowest) score.

While this invention has been described in terms of several preferredembodiments, there are alterations, permutations, and equivalents whichfall within the scope of this invention. It is therefore intended thatthe following appended claims be interpreted as including all suchalterations, permutations, and equivalents as fall within the truespirit and scope of the present invention.

What is claimed is:
 1. A method for recognizing handwritten entries on adisplay screen of a computer system, the computer system also having aprocessor and memory, the method comprising the steps of:receiving ahandwritten stroke written on the screen of said computer system, saidstroke being received in a form that can be manipulated by saidprocessor; determining a start point and a stop point of said stroke;normalizing said stroke with said processor by rotating said strokeabout a stroke center until at least one of said start point or saidstop point aligns with a predefined axis, thereby obtaining a normalizedstroke; matching said normalized stroke against a template of characterparts stored in memory to create one or more associated character partinterpretations; using said processor to group said normalized stroke,if possible, with one or more normalized strokes that were matched in alike fashion to create a character part group; and using said processorto recognize a higher level object from said character part grouputilizing the angles between the strokes of the character part group andusing said character part interpretations.
 2. A method as recited inclaim 1 wherein said higher level object is a character.
 3. A method asrecited in claim 1 wherein said higher level object is a word.
 4. Amethod as recited in claim 3 wherein entries in said template ofcharacter parts have been pre-rotated to match the normalized positionof said character part.
 5. A method as recited in claim 3 wherein saidstep of using a processor to group said normalized strokes groups thosestrokes that are related in space and time.
 6. A method as recited inclaim 3 wherein said step of recognizing a higher level object comprisesclassifying said character part group to provide a plurality of weightedinterpretations, and choosing the best weighted interpretation.
 7. Amethod for the recognition of handwritten objects on a display screen ofa computer system, the computer system also having a processor and amemory, the method comprising the steps of:receiving a handwrittenstroke written on the display screen of said computer system, saidstroke being received in a form that can be manipulated by saidprocessor; obtaining a start point and a stop point of said stroke;normalizing said stroke by rotating said stroke about a stroke centeruntil at least one of said start point or said stop point aligns with avertical axis, whereby a normalized stroke is obtained; matching saidnormalized stroke against a stroke database stored in memory to obtainat least one character part interpretation; and using said processor torecognize a handwritten object using one or more of said character partinterpretations.
 8. A method as recited in claim 7 further comprisingthe step of filtering said handwritten stroke before normalizing saidstroke.
 9. A method as recited in claim 7 wherein said step ofnormalizing said stroke comprises rotating said stroke by a predefinedangle to create a normalized stroke.
 10. A method as recited in claim 9wherein said step of normalizing said stroke further comprises the stepof normalizing an aspect ratio of said stroke.
 11. A method as recitedin claim 9 wherein at least one of said starting point and said endingpoint are at a predetermined position after said rotating step.
 12. Amethod as recited in claim 9, further comprising the step of creating astroke template database having a plurality of stroke template entriesagainst which said normalized stroke may be matched by said matchingstep.
 13. A method as recited in claim 12 wherein said stroke templateentries have been normalized to match the normalization of saidnormalized stroke.
 14. A method as recited in claim 13 furthercomprising the step of creating a character description database, andthe step of creating pointers from said stroke template entries to saidcharacter description database.
 15. A method as recited in claim 14further comprising the step of creating a character part descriptiondatabase, and the step of creating pointers from said characterdescription database to said character part description database.
 16. Amethod as recited in claim 9 wherein said step of recognizing ahandwritten object utilizes said stroke angles to help provide weightsfor multiple interpretations for said object.
 17. An apparatus for theangle-independent recognition of handwritten objects comprising:adisplay screen; a central processing unit (CPU); pointer means forwriting a handwritten stroke on said display screen; means for providingthe handwritten stroke to said CPU; means for demarcating a start pointand a stop point of said stroke: means for normalizing said stroke byrotating said stroke until one of said start point or said stop pointaligns with a predefined axis; means for matching said normalizedstrokes against a stroke database stored in memory to obtain a pluralityof character part interpretations; and means for recognizing, with saidCPU, a handwritten object using said character part interpretations. 18.An apparatus as recited in claim 17 wherein said means for normalizingstrokes comprises:means for detecting stroke angles of said strokesrelative to said predefined axis with said CPU; and means for rotatingsaid strokes with said CPU by said stroke angles to create normalizedstrokes.
 19. An apparatus as recited in claim 18 wherein said means fornormalizing strokes further comprises:means for detecting the aspectratios of said strokes; and means for normalizing said aspect ratios ofsaid strokes if said aspect ratios are not within predetermined limits.20. An apparatus as recited in claim 18 wherein said means for matchingsaid normalized strokes includes a stroke template database having aplurality of stroke template entries corresponding to a plurality ofcharacter parts.
 21. An apparatus as recited in claim 20 wherein saidplurality of stroke template entries are normalized to match theorientation of said plurality of strokes.
 22. An apparatus as recited inclaim 18 wherein said means for recognizing comprises:controller means;a plurality of recognition domains coupled to said controller means; andan arbiter coupled to said controller means for resolving multipleinterpretations of said object.
 23. An apparatus as recited in claim 22wherein said plurality of recognition domains includes a character partrecognition domain and a word recognition domain.
 24. A method for therecognition of handwritten objects on a display screen of a computersystem, the computer system also having a processor and a memory, themethod comprising the steps of:receiving a handwritten stroke written onthe display screen of said computer system, said stroke being receivedin a form that can be manipulated by said processor; normalizing thestroke by adjusting the aspect ratio of a bounding box surrounding thestroke when the aspect ratio is greater than a first predetermined valveor less than a second predetermined value, said step of normalizingreducing recognition errors due to distortion; normalizing said strokeby a process including the steps of:(i) identifying the endpoints of thestroke (ii) identifying two of the stroke's most widely spaced points,at least one of which is an endpoint of the stroke, (iii) defining aline between these points, (iv) determining an angle between the definedaxis and the line defined in step (iii), and (v) rotating the stroke bythe angle determined in step (iv) such that at least one of the stroke'sendpoints lies on said defined axis; matching the normalized strokeagainst a stroke database stored in memory to obtain at least onecharacter part interpretation; and using the processor to recognize ahandwritten object using one or more of the character partinterpretations.
 25. A method as recited in claim 24 wherein the firstpredetermined value is about 3 and the second predetermined value isabout one-third.
 26. A method as recited in claim 24 wherein the step ofnormalizing said stroke further comprises rotating the stroke by apredefined angle.
 27. A method for recognizing handwritten entries on adisplay screen of a computer system, the computer system also having aprocessor and memory, the method comprising the followingsteps:receiving a handwritten stroke written on the display screen ofsaid computer system, said stroke being received in a form that can bemanipulated by said processor; normalizing said stroke by a processincluding the steps of:(i) identifying the endpoints of the stroke (ii)identifying two of the stroke's most widely spaced points, at least oneof which is an endpoint of the stroke, (iii) defining a line betweenthese points, (iv) determining an angle between the defined axis and theline defined in step (iii), and (v) rotating the stroke by the angledetermined in step (iv) such that at least one of the stroke's endpointslies on said defined axis; matching said normalized stroke against astroke database stored in memory to obtain at least one character partinterpretation; and using said processor to recognize a handwrittenobject using one or more of said character part interpretations, whereinsaid step of normalizing reduces recognition errors due to handwritingwritten at various angles.
 28. The method or claim 27 further comprisinga step of identifying the center of said stroke.
 29. The method of claim28 wherein the step of rotating the stroke by said angle comprisesrotating the stroke about said center of the stroke.